The Minimum Variance Unbiased Estimator

This module motivates and introduces the minimum variance unbiased estimator (MVUE). This is the primary criterion in the classical (frequentist) approach to parameter estimation. We introduce the concepts of mean squared error (MSE), variance, bias, unbiased estimators, and the bias-variance decomposition of the MSE. The Minimum Variance Unbiased Estimator 1 In Search of a Useful Criterion In parameter estimation, we observe an N -dimensional vector X of measurements. The distribution of X is governed by a density or probability mass function f θ (x ), which is parameterized by an unknown parameter θ. We would like to establish a useful criterion for guiding the design and assessing the quality of an estimator ˆ θ (x). We will adopt a classical (frequentist) view of the unknown parameter: it is not itself random, it is simply unknown. One possibility is to try to design an estimator that minimizes the mean-squared error, that is, the expected squared deviation of the estimated parameter value from the true parameter value. For a scalar parameter, the MSE is de ned by MSE ( θ̂, θ ) = E [( ˆ θ (x)− θ )2] (1) For a vector parameter θ, this de nition is generalized by MSE ( θ̂, θ ) = E [( ˆ θ (x)− θ )T ( ˆ θ (x)− θ)] (2) The expectation is with respect to the distribution of X. Note that for a given estimator, the MSE is a function of θ. While the MSE is a perfectly reasonable way to assess the quality of an estimator, it does not lead to a useful design criterion. Indeed, the estimator that minimizes the MSE is simply the estimator ˆ θ (x) = θ (3) ∗http://creativecommons.org/licenses/by/1.0 http://cnx.org/content/m11426/latest/ Connexions module: m11426 2 Unfortunately, this depends on the value of the unknown parameter, and is therefore not realizeable! We need a criterion that leads to a realizeable estimator. Note: In the Bayesian Approach to Parameter Estimation1, the MSE is a useful design rule. 2 The Bias-Variance Decomposition of the MSE It is possible to rewrite the MSE in such a way that a useful optimality criterion for estimation emerges. For a scalar parameter θ, [Insert 1] This expression is called the bias-variance decomposition of the mean-squared error. The rst term on the right-hand side is called the variance of the estimator, and the second term on the right-hand side is the square of the bias of the estimator. The formal de nition of these concepts for vector parameters is now given: Let θ̂ be an estimator of the parameter θ. De nition 1: variance The variance of θ̂ is [Insert 2] De nition 2: bias The bias of θ̂ is [Insert 3] The bias-variance decomposition also holds for vector parameters: [Insert 4] The proof is a straighforward generalization of the argument for the scalar parameter case. Exercise 1: Prove the bias-variance decomposition of the MSE for the vector parameter case. 3 The Bias-Variance Tradeo The MSE decomposes into the sum of two non-negative terms, the squared bias and the variance. In general, for an arbitrary estimator, both of these terms will be nonzero. Furthermore, as an estimator is modi ed so that one term increases, typically the other term will decrease. This is the so-called bias-variance tradeo . The following example illustrates this e ect. Example 1: Let à = α 1 N ∑N n=1 xn, where xn = A+wn, wn ∼ N ( 0, σ ) , and α is an arbitrary constant. Let's nd the value of α that minimizes the MSE. MSE ( à ) = E [( Ã−A )2] (4) note: à = αSN , SN ∼ N ( A, σ 2 N ) MSE ( à ) = E [ à ] − 2E [ à ] A + A = αE [ 1 N2 ∑N ij=1 (xixj) ] − 2αE [ 1 N ∑N n=1 xn ] A + A = α 1 N2 ∑N ij=1 (E [xixj ])− 2α 1 N ∑N n=1 (E [xn]) +A 2 (5) http://cnx.rice.edu/content//latest/ http://cnx.org/content/m11426/latest/ Connexions module: m11426 3 E [xixj ] = { A + σ if i = j A if i 6= j MSE ( à ) = α ( A + σ 2 N ) − 2αA + A = α 2σ2 N + (α− 1) 2 A (6) σ ( à )2 = ασ N Bias ( à ) = (α− 1)A2 ∂ ∂α MSE ( à ) = 2ασ N + 2 (α− 1)A = 0 α∗ = A A2 + σ2 N (7) The optimal value α∗ dpends on the unknown parameter A! Therefore the estimator is not realizable. Note that the problematic dependence on the parameter enters through the Bias component of the MSE. Therefore, a reasonable alternative is to constrain the estimator to be unbiased, and then nd the estimator that produces the minimum variance (and hence provides the minimum MSE among all unbiased estimators). note: Sometimes no unbiased estimator exists, and we cannot proceed at all in this direction. In this example, note that as the value of α varies, one of the squared bias or variance terms increases, while the other one decreases. Futhermore, note that the dependence of the MSE on the unknown parameter is manifested in the bias. 4 Unbiased Estimators Since the bias depends on the value of the unknown parameter, it seems that any estimation criterion that depends on the bias would lead to an unrealizable estimator, as the previous example (Example 1) suggests (although in certain cases realizable minimum MSE estimators can be found). As an alternative to minimizing the MSE, we could focus on estimators that have a bias of zero. In this case, the bias contributes zero to the MSE, and in particular, it does not involve the unknown parameter. By focusing on estimators with zero bias, we may hope to arrive at a design criterion that yields realizable estimators. De nition 3: unbiased An estimator θ̂ is called unbiased if its bias is zero for all values of the unknown parameter. Equivalently, [Insert 5] For an estimator to be unbiased we require that on average the estimator will yield the true value of the unknown parameter. We now give some examples. The sample mean of a random sample is always an unbiased estimator for the mean. Example 2: Estimate the DC level in the Guassian white noise. Suppose we have data x1, . . . , xN and model the data by ∀n, n ∈ {1, . . . , N} : xn = A + wn http://cnx.org/content/m11426/latest/ Connexions module: m11426 4 where A is the unknown DC level, and wn ∼ N ( σ, σ ) . The parameter is −∞ < A <∞. Consider the sample-mean estimator: